Aerial video geo-registration using terrain models from dense and coherent stereo matching

In the context of aerial imagery, one of the first steps toward a coherent processing of the information contained in multiple images is geo-registration, which consists in assigning geographic 3D coordinates to the pixels of the image. This enables accurate alignment and geo-positioning of multiple images, detection of moving objects and fusion of data acquired from multiple sensors. To solve this problem there are different approaches that require, in addition to a precise characterization of the camera sensor, high resolution referenced images or terrain elevation models, which are usually not publicly available or out of date. Building upon the idea of developing technology that does not need a reference terrain elevation model, we propose a geo-registration technique that applies variational methods to obtain a dense and coherent surface elevation model that is used to replace the reference model. The surface elevation model is built by interpolation of scattered 3D points, which are obtained in a two-step process following a classical stereo pipeline: first, coherent disparity maps between image pairs of a video sequence are estimated and then image point correspondences are back-projected. The proposed variational method enforces continuity of the disparity map not only along epipolar lines (as done by previous geo-registration techniques) but also across them, in the full 2D image domain. In the experiments, aerial images from synthetic video sequences have been used to validate the proposed technique

Robust Image Registration with Global Intensity Transformation

This paper presents a registration method for images with global illumination variations. The method is based on a joint iterative optimization (geometric and photometric) of the L1 norm of the intensity error. Two strategies are compared to directly find the appropriate intensity transformation within each iteration: histogram specification and the solution obtained by analyzing the necessary optimality conditions. Such strategies reduce the search space of the joint optimization to that of the geometric transformation between the images

Augmented reality over video stream acquired from UAVs for operations support

La realidad aumentada (RA) se ha convertido, g los últimos avances tecnológicos, en una de las disciplinas con mayor crecimiento. El potencial de la RA propicia su estudio, no sólo para nuevos dispositivos específicos como gatas y cascos, sino para cualquier dispositivo dotado de una cámara. Con esta idea, Airbus impulsó un proyecto de innovación, Situational Awareness Virtual EnviRonment (SAVIER), para incorporar la RA a sus estaciones de tierra, mejorando así el flujo de vídeo capturado desde la cámara de sus vehículos aéreos no tripulados (UAV). Esta tesis se enmarca en ese proyecto y explora distintas estrategias para mejorar la conciencia de situación de los operadores de UAV durante el transcurso de una misión. Inicialmente, la tesis aborda el geo-registro, que es una estrategia utilizada para localizar al UAV en zonas sin acceso a señal de posicionamiento global (GPS). Esto tiene interés porque conocer la posición del UAV es esencial para poder proporcionar información sobre los alrededores. Por ello, la tesis propone dos sistemas clave para el geo-registro, atendiendo a los diferentes datos de referencia que se utilicen. En primer lugar, un esquema de procesado estéreo multi-vista para construir un modelo de terreno denso a partir de imágenes del vídeo capturado por el UAV. Es útil cuando se necesita como referencia un modelo del terreno pero no está disponible, está desactualizado o tiene baja resolución. El método variacional propuesto impone continuidad no sólo a lo largo de la línea epipolar sino también transversalmente, en todo el dominio de la imagen. En segundo lugar, la tesis propone un método conjunto (geométrico y fotométrico) de registro de imágenes que puede lidiar con tipos de distorsión genéricos: deformaciones parametrizadas (como homografías) y transformaciones fotométricas no lineales. Es un método de registro basado en zonas de las imágenes, que permite operar en escenarios donde los métodos de geo-registro basados en puntos característicos no son fiables. Por último, se considera el caso general, donde todas las medidas de los sensores tienen suficiente precisión y la tesis se centra en mostrar elementos virtuales sobre el flujo de vídeo. Se desarrolla una herramienta de RA para mejorar la conciencia de situación de operadores de UAV durante misiones de inteligencia y vigilancia. El sistema de RA proporciona información sobre la ruta de vuelo y los objetivos; de esta forma el operador puede reducir el tiempo de búsqueda para encontrarlos incluso si están ocultos. La usabilidad de la herramienta propuesta se demostró con la adopción de estándares de la OTAN y fue plenamente integrada en el demostrador de SAVIER de Airbus, en Getafe, Madrid. ----------ABSTRACT---------- Augmented reality (AR) has become, due to recent technology developments, a fastgrowing discipline. The potential of AR supports its study not only for specific devices such as glasses or helmets, but for anything equipped with a camera. Following this idea, Airbus promoted an innovation project, Situational Awareness Virtual EnviRonment (SAVIER), to incorporate AR in their ground control stations, thus allowing the enhancement of the video stream captured from Unmanned Aerial Vehicles (UAVs). This thesis is framed in that project and explores different approaches to improve the situational awareness of the UAV operators during a mission. Initially, the thesis is focused on geo-registration, a strategy used for the localization of the UAV in GPS-denied environments. This is of interest because knowing the position of the UAV is essential to provide information about the surroundings. For this reason, we proposed two key systems for geo-registration with different reference data. First, a multi-view stereo processing pipeline for building a dense terrain model from images of the UAV video feed. This is helpful when a reference terrain model is needed for geo-registration but it is unavailable, outdated, or it has low resolution. The proposed variational method enforces continuity not only along epipolar lines but also across them, in the full image domain. Second, the thesis proposed a joint geometric and photometric image registration method that can deal with generic types of distortion: parametric warpings (such as homographies) and non-linear photometric transformations. It is built on top of area-based registration methods to be able to operate in scenarios where feature-based geo-registration methods are not reliable. Finally, the general case was considered, in which every sensor measurement is known with enough accuracy and the thesis focused on displaying virtual elements over the video stream acquired by the UAV. An AR tool to improve the situational awareness of UAV operators during intelligence and surveillance missions was developed. The AR system provides information about the flying path and the targets, so that the operator can reduce the time to find them even in the presence of occlusions. The usability of the proposed AR tool was proved by the adoption of NATO standards and it was fully integrated with the Airbus SAVIER demonstrator, in Getafe, Madrid